fix: flush OTEL spans between sequential studies and fix transform defaults#33
Merged
Merged
Conversation
…faults Two bugs fixed: 1. OTEL span loss in multi-study workflows: When running N+1 transform comparisons, multi-attack campaigns, or category sweeps, only the first study's traces were reliably exported to the platform. The BatchSpanProcessor buffers spans on a background thread — later studies' spans were still in the buffer when dn.shutdown() raced against process teardown. Added explicit force_flush() between sequential assessment.run() calls in all 3 multi-study templates. 2. Unwanted baseline run: When user specified transforms (e.g. 'run crescendo with Telugu'), the agent was forced to set compare_transforms=true, creating an N+1 study with an unrequested baseline run. Changed default to compare_transforms=false — transforms are now applied directly. N+1 comparison only triggers when user explicitly asks to 'compare' or 'benchmark' transforms. Bump version: 1.3.2 → 1.3.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bugs fixed in the AI Red Teaming capability.
Bug 1: OTEL span loss in multi-study workflows
When running N+1 transform comparisons, multi-attack campaigns, or category sweeps, only the first study's traces appeared on the platform. The
BatchSpanProcessorbuffers spans on a background thread — later studies' spans were still in the buffer whendn.shutdown()raced against process teardown.Fix: Added explicit
force_flush(timeout_millis=10_000)between sequentialassessment.run()calls in all 3 multi-study templates:_TRANSFORM_STUDY_TEMPLATE(N+1 transform comparisons)_CAMPAIGN_ATTACK_BLOCK(multi-attack campaigns)Bug 2: Unwanted baseline run with transforms
When user said "run crescendo with Telugu", the agent forced
compare_transforms=true, creating an N+1 study with an unrequested baseline run. The user only wanted the transform applied.Fix: Changed default to
compare_transforms=false. Transforms are applied directly — no baseline added. N+1 comparison only triggers when user explicitly asks to "compare" or "benchmark" transforms.Version bump: 1.3.2 → 1.3.5
Files changed (3 files, +40 / -4)
agents/ai-red-teaming-agent.mdcompare_transforms=falseunless user asks to comparescripts/attack_runner.pyforce_flush()between sequential studies in 3 templatescapability.yamlTesting
adapt_language(Telugu)— only baseline appeared on platform